Information-inputting device inputting contact point of object on recording surface as information

ABSTRACT

Structure and function for inputting information preferably includes a display device having two cameras in respective corners thereof. At least one computer readable medium preferably has program instructions configured to cause at least one processing structure to: (i) extract an object located on a plane of the display device from an image that includes the plane of the object, (ii) determine whether the object is a writing implement by determining, when a plurality of objects are extracted from the image, that one of the plurality of objects that satisfies a prescribed condition is the writing implement, (iii) calculate a position of a contact point between the writing implement and the plane as information to be input if the object has been determined as the writing implement, and (iv) input the information representing a position on the plane indicated by the object.

This application is a continuation of U.S. patent application Ser. No.12/788,822, filed May 27, 2010 (now U.S. Pat. No. Re. 42,794, issuedOct. 4, 2011), which is a reissue of U.S. patent application Ser. No.10/976,990, filed Nov. 1, 2004, now U.S. Pat. No. 7,379,622, which is adivisional of U.S. patent application Ser. No. 09/742,063, filed Dec.22, 2000, now U.S. Pat. No. 6,829,372, the contents of all areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information-inputting device thatcan record data in a recording medium in real time while the data isbeing written on a recording surface without specifying types of writingimplements and materials used for the recording surface.

2. Description of the Related Art

Methods and apparatuses for inputting information have been proposed,and have been put to practical use. For instance, Japanese Laid-openPatent Application No. 6-289989 discloses an information recognitiondevice. The information recognition device provides a camera on one sideof a recording surface. The information recognition device detectscoordinates of each position of information recorded by a writingimplement on the recording surface, for example, a track of letterswritten on recording paper, and develops depiction data based on thecoordinates. The information recognition device alternatively detectscoordinates of each position of a writing implement, that is,coordinates of a tip of the writing implement, and develops depictiondata based on the coordinates. To recognize the information recorded onthe recording surface, the information recognition device also needs todetect the origin of the coordinates on the recording surface bydetecting corners of the recording paper, that is, the recordingsurface, and the origin of the X-axis and the Y-axis recorded on therecording paper.

The information recognition device must detect the corners of therecording paper or the origin of the coordinates based on theinformation recorded on the recording paper before detecting thecoordinates of each position of the information every time the recordingpaper is exchanged. Additionally, in a case that a position of therecording paper is moved from its original position while recording theinformation on the recording paper, or in a case that positions of thecorners of the recording paper in an image being taken by the camera aremoved. coordinates of recording the information on the recording paperare detected by the camera as different coordinates from what they aresupposed to be, since the coordinates of recording the information arecalculated based on the origin of the coordinates. Consequently, thecamera detects different information from what a user intends to recordunless a moved origin of the coordinates is detected.

Additionally, when the camera photographs the information recorded onthe recording paper from an upper oblique direction of the recordingpaper, error between actual coordinates of the information recorded onthe recording paper and coordinates detected by the camera increases asa distance between the recording paper and the camera increases. Such aproblem will be described with reference to FIGS. 1A and 1B. An angle θindicates an angle range that a fixed number of pixels provided in thevertical direction of an imaging device such as the camera can detectlight through a lens. In FIG. 1A, a distance on the recording paper thatthe imaging device can detect with the angle θ is a distance L1. In FIG.1B, a distance on the recording paper that the imaging device can detectwith the angle θ is a distance L2. It is assumed that the distance L2 isgreater than the distance L1. Accordingly, the greater the distancebetween the recording paper and the camera, the wider an area on therecording paper that can be detected by the camera with the angle θ. Inother words, an area on the recording paper detected by each pixel thatis provided in the imaging device increases as the distance between therecording paper and the camera increases. Accordingly, when the cameraphotographs the information recorded on the recording paper from theupper oblique direction of the recording paper, the error between theactual coordinates of the information recorded on the recording paperand the coordinates detected by the camera increases as the distancebetween the recording paper and the camera increases. Thus, an imagedetected by the camera is contracted with respect to an imagecorresponding to the information recorded on the recording paper.

The number of pixels provided in the imaging device necessary forobtaining image data by photographing the coordinates of each positionof the information depends on a size of a recording area on therecording paper and a resolution of reading the coordinates of eachposition of the information recorded on the recording paper. As the sizeof the recording area increases, the greater the number of pixelsnecessary for calculating the coordinates from the image data alsoincreases. Additionally, for a higher resolution of reading thecoordinates, the number of pixels must be greater. Furthermore, a framerate of image signals outputted from the imaging device must be high inorder to clearly monitor an information recording process from thephotographed image data. However, an imaging device with a large numberof pixels and a high frame rate is generally expensive, and thus it ishard to hold down production cost of an information input device bymounting such an imaging device thereon. Consequently, a writing inputdevice with a comparatively inexpensive imaging device using a smallernumber of pixels has been requested.

Additionally, a size of an image-display device used in a portablewriting input device for displaying the image data obtained by thecamera is preferred to be small for miniaturization of the device andits electric efficiency. However, if the size of the image-displaydevice is small, the number of pixels displayed on the image-displaydevice becomes small, and thus quality of the image data displayed onthe image-display device decreases by stretching the image dataimmoderately when displaying a page of the image data thereon.Furthermore, when the number of imaging devices used for photographingthe information recorded on the recording paper is small, a wide-anglelens should be attached to each of the imaging devices for photographingthe information. In such case, a resolution of reading the coordinatesof the information recorded on the recording paper differs depending onwhere the information is recorded on the recording paper.

SUMMARY OF THE INVENTION

Accordingly, it is a general object of the present invention to providean information-inputting device used for writing data on a recordingsurface by use of a writing implement. A more specific object of thepresent invention is to provide an information-inputting device, awriting input device and a portable electronic writing input device thatare easily carried and are used for writing data on a desired recordedmaterial whose surface is a plane by use of a desired writing implement.Another object of the present invention is to provide a method and anapparatus for recording data in a recording medium in real time whilethe data is being written on a recording surface by use of a writingimplement in a case in which the writing implement is detected. Yet,another object of the present invention is to provide a method ofmanaging written data and a recording medium for creating a page of thewritten data by dividing the written data into a plurality of parts andby inputting the plurality of parts by a recording area in a case inwhich a resolution of reading the written data in the recording area islow. Yet, another object of the present invention is to provide a methodof managing written data and a recording medium for controlling a pixeldensity of the written data to be even throughout an entire recordingarea. Yet, another object of the present invention is to provide amethod of controlling display of a recording area to increaseoperability of writing data in the recording area.

The above-described objects of the present invention are achieved by aninformation-inputting device including a plurality of photographingunits photographing an area on a plane; an object-recognizing unitextracting an object located on the plane from a photographed image, andrecognizing whether the object is a specific object; alocation-calculating unit calculating a contact position of the specificobject on the plane from the photographed image if the object has beenrecognized as the specific object; and a data-storing unit storinginformation about a track of the contact position while the specificobject is contacting the plane.

The above-described objects of the present invention are also achievedby a method of inputting information including the steps of extractingan object located on a plane from an image that includes the plane andthe object; recognizing whether the object is a specific object; andinputting a contact position of the specific object on the plane asinformation if the object has been recognized as the specific object.

The above-described objects of the present invention are also achievedby a writing input device including an image-inputting unitphotographing a recording area on a plane by providing a plurality ofelectronic cameras that include imaging devices; an object-recognizingunit extracting an object located on the plane from a photographedimage, and determining whether the object is a writing implement byrecognizing a shape of the object; a coordinate-calculating unitcalculating contact coordinates of the object on the plane based on animage of the object on an imaging device if the object has beendetermined as the writing implement; a data-storing unit storing aseries of the contact coordinates while the object is contacting theplane; and a displaying unit creating depiction data from the series ofthe contact coordinates, and displaying the depiction data thereon.

The above-described objects of the present invention are also achievedby a method of managing written data in a writing input device, whereinthe writing input device includes an image-inputting unit photographinga recording area on a plane by providing a plurality of electroniccameras that includes imaging devices; an object-recognizing unitextracting an object located on the plane from a photographed image, anddetermining whether the object is a writing implement by recognizing ashape of the object; a coordinate-calculating unit calculating contactcoordinates of the object on the plane based on an image of the objecton an imaging device if the object has been determined as the writingimplement; a data-storing unit storing a series of the contactcoordinates while the object is contacting the plane; and a displayingunit creating depiction data from the series of the contact coordinates,and displaying the depiction data thereon, the method including thesteps of dividing a page of a data area into a plurality of areas;assigning one of the areas to the recording area; and managing thewritten data to be recorded in the recording area as data of the one ofthe areas in the page.

The above-described objects of the present invention are also achievedby a portable electronic writing input device including a main bodyunit; a first camera unit; a second camera unit; and anexpansion/contraction unit connecting the first and second camera unitson left and right parts of the main body unit as well as expanding orcontracting an interval between the main body unit and the first orsecond camera unit, wherein the portable electronic writing inputdevice, being placed on a plane material, photographs a movement of awriting implement by use of the first and second camera units, when auser writes data on the plane material by using the writing implement.

The above-described objects of the present invention are also achievedby a recording medium readable by a computer, tangibly embodying aprogram of instructions executable by the computer to perform a method,the method including the steps of extracting an object located on aplane from an image including the object and the plane; recognizingwhether the object is a specific object; calculating a contact positionof the specific object on the plane if the object has been recognized asthe specific object; storing written data including a series ofcoordinates of the contact position calculated while the object iscontacting the plane; generating depiction data from the written data;and displaying the depiction data.

The information-inputting device determines a shape of the object on theplane by use of the object-recognizing unit, and calculates the contactpositions of the specific object by use of the location-calculating unitif the object has been determined as the specific object by theobject-recognizing unit. Subsequently, the information-inputting devicestores the track of the contact position while the specific object iscontacting the plane. Thus, the information-inputting device can recorddata in a recording medium in real time while the data is being writtenon a recording surface without specifying types of writing implementsand materials used for the recording surface.

Other objects, features and advantages of the present invention willbecome more apparent from the following detailed description when readin conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are diagrams used for describing a situation in which anerror between actual coordinates of information recorded on recordingpaper and coordinates detected by a camera increases as distance betweenthe recording paper and the camera increases when the camera photographsthe information recorded on the recording paper from an upper obliquedirection of the recording paper;

FIG. 2 is a diagram showing an appearance of a writing input deviceaccording to the present invention;

FIG. 3 is a block diagram showing a structure of a camera unit providedin the writing input device;

FIG. 4 is a block diagram showing a recording area where the writinginput device can obtain coordinates of information recorded thereon;

FIG. 5 is a block diagram showing a system structure of the writinginput device according to the present invention;

FIG. 6 is a diagram for describing a method to obtain coordinates of acontact point of a writing implement contacting the recording area;

FIG. 7 is a diagram showing an enlargement of an area where a wide-anglelens and a CMOS image sensor are placed in FIG. 6;

FIG. 8 is a flowchart describing steps performed by the writing inputdevice according to a first embodiment of the present invention;

FIG. 9 is a diagram showing an image wherein a standard line is rotatingaround center of gravity of the writing implement;

FIG. 10 is a diagram showing an image of the writing implement havingsymmetry with respect to an axis of symmetry;

FIG. 11 is a diagram showing an image wherein the writing implement iscontacting a recording surface;

FIG. 12 is a flowchart describing steps performed by the writing inputdevice according to a second embodiment of the present invention;

FIG. 13 is a flowchart describing steps performed by the writing inputdevice according to a third embodiment of the present invention;

FIGS. 14A, 14B and 14C are diagrams showing icons corresponding totypical writing implements;

FIG. 15 is a flowchart describing steps performed by the writing inputdevice according to a fourth embodiment of the present invention;

FIG. 16 is a flowchart describing steps performed by the writing inputdevice according to a fifth embodiment of the present invention;

FIG. 17 is a diagram showing a case in which the writing input devicerecognizes a plurality of writing implements;

FIG. 18 is a flowchart describing steps performed by the writing inputdevice according to a sixth embodiment of the present invention;

FIG. 19 is a diagram showing a method of obtaining a valid contact pointfrom a plurality of contact points when a plurality of writingimplements are contacting the recording surface;

FIG. 20 is a flowchart describing steps performed by the writing inputdevice according to a seventh embodiment of the present invention;

FIG. 21 is a flowchart describing steps performed by the writing inputdevice according to an eighth embodiment of the present invention;

FIG. 22 is a flowchart describing steps performed by the writing inputdevice according to a ninth embodiment of the present invention;

FIG. 23 is a diagram showing the recording area set by a user operation;

FIG. 24 is a flowchart describing steps performed by the writing inputdevice according to a tenth embodiment of the present invention;

FIG. 25 is a block diagram showing the recording area created by use ofa frame;

FIG. 26 is a diagram used for describing that a resolution of readinginformation written in the recording area varies depending on a distancefrom the camera unit;

FIG. 27 is a block diagram showing a method of dividing a page of a dataarea into displaying blocks;

FIG. 28 is a block diagram showing a method of dividing a page of thedata area into writing blocks;

FIG. 29 is a block diagram showing a relation between a page of the dataarea and the recording area;

FIG. 30 is a flowchart describing steps performed by the writing inputdevice according to a thirteenth embodiment of the present invention;

FIG. 31 is a block diagram showing a method of assigning apixel-displaying area and a displaying area for writing to desiredlocations in a page of the data area;

FIGS. 32A and 32B are diagrams showing an accentuated displaying areafor writing;

FIG. 33 is a flowchart describing steps performed by the writing inputdevice according to a sixteenth embodiment of the present invention;

FIG. 34 is a diagram showing a location of a writing block displayed ona LCD;

FIG. 35 is a flowchart describing steps performed by the writing inputdevice according to a seventeenth embodiment of the present invention;

FIG. 36 is a diagram showing the writing block shown in FIG. 34 afterbeing moved by use of a direction-specifying key;

FIG. 37 is a flowchart describing steps performed by the writing inputdevice according to an eighteenth embodiment of the present invention;

FIG. 38 is a block diagram showing the pixel-displaying area and asummarized location of the pixel-displaying area in a page of the dataarea;

FIG. 39 is a block diagram showing the displaying area for writing afterbeing moved from a position shown in FIG. 38 to a left edge of thepixel-displaying area by use of the direction-specifying key;

FIG. 40 is a diagram used for describing a method of calculating aresolution of reading coordinates in the recording area;

FIGS. 41A and 41B are diagrams showing directions of writing data;

FIG. 42 is a flowchart describing steps performed by the writing inputdevice according to a nineteenth embodiment of the present invention;and

FIG. 43 is a block diagram showing a system wherein the presentinvention is implemented by use of software.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A description will now be given of preferred embodiments of the presentinvention, with reference to the accompanying drawings.

FIG. 2 is a diagram showing an appearance of a writing input deviceaccording to the present invention. A writing input device 1 shown inFIG. 2 includes a left camera unit 2, a right camera unit 3,photographing windows 4 and 5, expansion/contraction units 6 and 7, anLCD (Liquid Crystal Display) 8, and LEDs (Light-Emitting Diodes) 9through 12. The writing input device 1 recognizes a shape of an objectfrom an object image photographed by the left camera unit 2 and theright camera unit 3, obtains coordinates of the object, and records thecoordinates of the object as data in a recording medium. The writinginput device 1 is portable, and is placed on a flat recording surfacesuch as a surface of a table or a surface of paper provided on thetable, as shown in FIG. 2. A recording area 13 facing the left cameraunit 2 and the right camera unit 3 is then formed. An optical axis oflight entering into the camera units 2 and 3 is parallel to therecording surface, and the light is inputted to the writing input device1 through the photographing windows 4 and 5, which are transparentplates. The light entering the writing input device 1 is photographed byan imaging device provided in the writing input device 1 after beingreflected by a mirror and passing through a wide-angle lens alsoprovided therein. In the present invention, a CMOS (Complementary MetalOxide Semiconductor) image sensor is used as the imaging device, forinstance.

The expansion/contraction units 6 and 7 are expanded and contracted byhand operations, and adjust a distance between the left camera unit 2and the right camera unit 3. The LCD 8 displays information such asletters that are written in the recording area 13. Additionally, the LCD8 includes various buttons used for its operations, but the buttons arenot shown in FIG. 2. The LED 9 is provided on an area above thephotographing window 4. The LED 10 is provided on an area above thephotographing window 5. The LED 11 and the LED 12 are provided on anarea below the LCD 8. The LEDs 9 though 12 are used for illuminating therecording area 13. The above-described LEDs 9 through 12 can be providedon other areas as long as they can illuminate the recording area 13.

FIG. 3 is a block diagram showing a structure of the left camera unit 2.FIG. 3 more particularly shows the structure of the left camera unit 2seen from the side. Both of the left camera unit 2 and the right cameraunit 3 have the same structure. The camera unit 2 includes a mirror 14,a wide-angle lens 15, a CMOS image sensor 16 and an image-processingcircuit 17, as shown in FIG. 3. Light with its optical axis parallel tothe recording surface enters the writing input device 1 through thephotographing window 4, and is reflected by the mirror 14 toward thewide-angle lens 15. The light then arrives at the CMOS image sensor 16through the wide-angle lens 15. The CMOS image sensor 16 executesphotoelectric conversion on the received light to create an electricsignal by rise of a photo-diode provided in each pixel. Subsequently,the CMOS image sensor 16 amplifies the electric signal by use of a cellamplifier used for each pixel, and transmits the amplified signal to theimage-processing unit 17 every time a fixed period passes.

FIG. 4 is a block diagram showing a recording area where the writinginput device 1 can obtain coordinates of information recorded thereon.FIG. 4 shows the writing input device 1 and the recording surface seenfrom above. The recording area 13 is shown in a shaded area in FIG. 4. Arecording position, that is, a position of a writing implement touchingthe recording surface, can be calculated by using trigonometry fromimages photographed by the left camera unit 2 and the right camera unit3, and thus the recording area 13 wherein coordinates of the recordingposition can be obtained is an area on the recording surface where aphotographing area of the left camera unit 2 and a photographing area ofthe right camera unit 3 overlap. In other words, an area surrounded bybroken lines is the recording area 13. A lower boundary of the recordingarea 13 shown in FIG. 4 is a lowest position of the writing implementwhere the left camera unit 2 and the right camera unit 3 can detect theposition of the writing implement. FIG. 4 also shows the mirror 14included in the left camera unit 2 and a mirror 19 included in the rightcamera unit 3. The mirror 14 is placed in the left camera unit 2 inorder to reflect incident light of about a 90-degree range entering fromthe photographing window 4, and to transmit the incident light to theCMOS image sensor 16 through the wide-angle lens 15. Similarly, themirror 19 is placed in the right camera unit 3 in order to reflectincident light having about 90 degrees range and entering from thephotographing window 5, and to irradiate the incident light to alater-mentioned CMOS image sensor 21 through a later-mentionedwide-angle lens 20.

FIG. 5 is a block diagram showing a system structure of the writinginput device 1. The writing input device 1 includes the left camera unit2, the right camera unit 3, a CPU (Central Processing Unit) 23 or aprocessing unit 23, a ROM (Read-Only Memory) 24, a main memory 25, aflash memory 26, an operation unit 27, an LCD display-control unit 28, aUSB (Universal Serial Bus) driver 29, a sound source 30, the LCD 8, aUSB interface (I/F) 31, a speaker 32, a DC-DC converter 33, a battery34, the LED 11, the LED 12, and a system bus 35. The left camera unit 2includes the mirror 14, the wide-angle lens 15, the CMOS image sensor16, the image-processing circuit 17, and the LED 9. The right cameraunit 3 includes the mirror 19, the wide-angle lens 20, the CMOS imagesensor 21, an image-processing circuit 22, and the LED 10.

The mirror 14 reflects incident light entering through the photographingwindow 4 toward the wide-angle lens 15. The wide-angle lens 15 having aview angle of 90 degrees is placed in the left camera unit 2 to transmitthe light reflected by the mirror 14 to the CMOS image sensor 16. TheCMOS image sensor 16 executes photoelectric conversion on the lightreceived through the wide-angle lens 15 to an analog signal by use of aphoto-diode provided for each pixel therein, amplifies the analog signalby using a cell amplifier provided for each pixel, and outputs theamplified analog signal to the image-processing unit 17 every time afixed period passes. Similarly, in the right camera unit 3, the mirror19 reflects incident light entering through the photographing window 5toward the wide-angle lens 20. The wide-angle lens 20 having a viewangle of 90 degrees is placed in the right camera unit 3 to transmit thelight reflected by the mirror 19 to the CMOS image sensor 21. The CMOSimage sensor 21 executes photoelectric conversion on the light receivedthrough the wide-angle lens 20 to an analog signal by use of aphoto-diode provided for each pixel therein, amplifies the analog signalby using a cell amplifier provided for each pixel, and outputs theamplified analog signal to the image-processing unit 22 every time afixed period passes.

Each of the image-processing circuits 17 and 22 includes an A/D(Analog/Digital) conversion circuit, and converts an analog signalrespectively received from the CMOS image sensors 16 and 21 to a digitalsignal. Subsequently, the image-processing circuits 17 and 22 execute aprocess to extract an outline of a subject image from image dataobtained by converting the analog signal to the digital signal, an imagerecognition process to decide whether the subject image is a writingimplement based on the extracted outline, and a process to outputinformation about a position where an object detected as the writingimplement is contacting the recording surface. It should be noted thatthe left camera unit 2 and the right camera unit 3 perform theabove-described processes synchronously with each other.

The ROM 24 initially stores a program to control the writing inputdevice 1. The main memory 25 includes a DRAM (Dynamic Random AccessMemory), and is used as a work area for the CPU 23. The flash memory 26stores coordinate data, that is, data about the coordinates of theinformation recorded in the recording area 13. The operation unit 27includes various types of keys near the LCD 8, the keys being used fordisplaying the coordinate data stored in the flash memory 26 on the LCD8, for forwarding the coordinate data to a personal computer through aUSB cable, and for other objects. The LCD display-control unit 28controls displaying of the coordinate data, an operation menu, and thelike on the LCD 8. The USB driver 29 transmits data to a device such asa personal computer connected to a USB cable, and receives data from thedevice, by executing operations based on a USB standard. The soundsource 30 generates a sound signal such as an alarm, and then thegenerated sound signal is outputted from the speaker 32. The battery 34is, for example, a nickel-metal hydride battery or a lithium battery. Anelectric current is supplied from the battery 34 through the DC-DCconverter 33 to units in the writing input device 1 in addition to theLEDs 9 through 12. The system bus 35 connects the image-processing units17 and 22, the CPU 23, the ROM 24, the main memory 25, the flash memory26, the operation unit 27, the LCD display-control unit 28, the USBdriver 29, and the sound source 30.

A description will now be given of a method to obtain coordinates of acontact point of a writing implement contacting the recording surface.As described with reference to FIG. 3, the CMOS image sensors 16 and 21are placed downwards respectively in the left camera unit 2 and theright camera unit 3. However, in order to describe the method, it isassumed in FIG. 6 that the mirrors 14 and 19 are omitted. Additionally,it is assumed that the wide-angle lenses 15 and 20, and the CMOS imagesensors 16 and 21 are placed as shown in FIG. 6 so that the optical axisof the incident light becomes parallel to the recording surface. Adistance between the wide-angle lenses 15 and 20 is named a distance L.The contact point of the writing implement is set as a contact point A.The coordinates of the contact point A are set to (x, y). A straightline connecting the wide-angle lenses 15 and 20 is named a X-Line. Anangle between the X-Line and a straight line drawn from the center ofthe wide-angle lens 15 to the contact point A is named an angle β1. Anangle between the X-Line and a straight line drawn from the center ofthe wide-angle lens 20 to the contact point A is named an angle β2. FIG.7 is a diagram showing an enlargement of an area where the wide-anglelens 15 and the CMOS image sensor 16 are placed in FIG. 6. In FIG. 7,distances “f” and “h” respectively indicate a distance between thewide-angle lens 15 and the CMOS image sensor 16, and a distance betweenan image-formed location of the optical axis 18 of the wide-angle lens15 and an image-formed location of the contact point A on the CMOS imagesensor 16. An angle α is an angle between the optical axis 18 of thewide-angle lens 15 and the X-Line. An angle θ is an angle between theoptical axis 18 of the wide-angle lens 15 and a straight line connectingthe contact point A and its image-formed location. The following twoequations are derived from FIGS. 6 and 7.θ=arctan(h/f)  EQ1β1=α−θ  EQ2

A value of the angle β1 for the left camera unit 2 can be obtained fromthe above equations since the angle α is initially measured as an anglefor positioning the wide-angle lenses 15 and 20 so that the optical axesof the wide-angle lenses 15 and 20 intersect with each other. Similarly,a value of the angle β2 can be obtained for the right camera unit 3.Once the values of the angles β1 and β2 have been obtained, thecoordinates of the contact point A (x, y) can be obtained by usingtrigonometry.x=L*tan β2/(tan β1+ tan β2)  EQ3y=x*tan β1  EQ4

Assuming the mirrors 14 and 19 do not exist in the left camera unit 2and the right camera unit 3, the description has been given of themethod to calculate the coordinates of the contact point A, that is, thecontact point of the writing implement. However, since the mirrors 14and 19 are provided in the writing input device 1 only for changing adirection of the optical axis placed in each of the left camera unit 2and the right camera unit 3 by reflecting light on the optical axis, theabove-described method can also be applied to calculate the coordinatesof the contact point of the writing implement in a case that there aremirrors placed in each of the left camera unit 2 and the right cameraunit 3. As describe above, both an optical axis of light emitted towardthe left camera unit 2 and an optical axis of light emitted toward theright camera unit 3 are parallel to a recording surface, and intersecteach other. Accordingly, both of the left camera unit 2 and the rightcamera unit 3 can detect coordinates of a contact point accurately withno errors. In addition, an area where coordinates of the contact pointare read by the left camera unit 2 and the right camera unit 3 can bekept wide.

A detailed description will now be given of embodiments based on thewriting input device 1 according to the present invention.

A first embodiment of the present invention relates to a method ofstoring information as digital data in a recording medium in real timewhile a user is recording the information by use of a pen on a recordedmaterial such as paper. The first embodiment additionally relates to amethod of storing information as digital data in a recording medium inreal time while a user is recording the information on the recordedmaterial by use of a writing implement such as a stick or a fingerwhereto ink is not applied. FIG. 8 is a flowchart describing stepsperformed by the writing input device 1 according to the firstembodiment of the present invention. The steps shown in FIG. 8 areactually performed by the image-processing circuits 17 and 22, and theCPU 23 provided in the writing input device 1. When a user records orwrites information on a recording surface of a recorded material such aspaper by use of a writing implement such as a pen, the left camera unit2 and the right camera unit 3 photograph writing movement performed bythe user. FIG. 9 shows an image of a writing implement 36 photographedby the left camera unit 2 and the right camera unit 3.

The image-processing circuit 17 of the left camera unit 2 converts ananalog image signal outputted from the CMOS image sensor 16 into adigital image signal in order to obtain a frame of image data, andextracts an outline of an object from the image data at a step S101. Ina case that the number of pixels of the CMOS image sensor 16 in avertical direction (an upward direction of a vertical side 39) is large,the image-processing circuit 17 controls outputting image Signals onlyfor pixels of the CMOS image sensor 16 that form an image having a fixedheight from a recording surface 38. An example of the above-describedmethod of extracting the outline of the object is to calculate a densitygradient between pixels by applying differentiation, and then to definethe outline based on the direction of increasing density gradient andthe size of the calculated density gradient. Such method is disclosed inJapanese Laid-open Patent Application No. 63-193282. After the outlinehas been extracted from the image data, the image-processing circuit 17determines whether the object is a writing implement or not based on ashape of the extracted outline of the object. An example of imagerecognition technology to determine the shape of the extracted outlineis to obtain a center of gravity 40 of the object, to calculatedistances between the center of gravity 40 and points on the outlinespecified by angles based on the center of gravity 40 in order, and thento define the shape of the extracted outline from relations between thedistances and the angles. Such a method is disclosed in JapaneseLaid-open Patent Application No. 8-315152. Subsequently, data about theshape of the outline obtained by the above-described method is comparedwith data that has been stored previously as shapes of writingimplements in the ROM 24 or in the flash memory 26 at a step S102.Consequently, the object is defined by the image-processing circuit 17either as a writing implement or as an object other than writingimplements at a step S103. If the object is detected as the object otherthan the writing implements, the image-processing circuit 17 proceeds tothe step S101.

Since an angle between the object and the recording surface 38 is notfixed, the data about the object is compared with the data stored in theROM 24 or the like by rotating a standard line 41 connecting the centerof gravity 40 of the object and the outline of the object in an range ofcertain angles as shown in FIG. 9 in order to define the shape of theobject. A plurality of shapes of writing implements can be provided asthe data stored in the ROM 24 or the like, wherein the data about theobject can be compared with each of the plurality of shapes of writingimplements. Another method of defining the object as a writing implementis to check whether the object is symmetric with respect to an axis ofsymmetry 42 shown in FIG. 10. When the object has been detected as asymmetrical object, the object can be defined as a writing implement.The symmetry of the object can be detected by calculating a distancefrom the center of gravity 40 to certain points on the outline locatedaround the center of gravity 40 in order.

If the object has been detected as a writing implement at the step S103,the image-processing circuit 17 determines whether the writing implementhas contacted the recording surface 38 as shown in FIG. 11 at a stepS104. It should be noted that the recording surface 38 corresponds to abottom side of a photographing area 37 shown in FIGS. 9, 10 and 11.Thus, to be concrete, the image-processing circuit 17 checks whether animage of the object in the photographing area 37 touches the bottom sidethereof. If it is determined at the step S104 that the writing implementhas not contacted the recording surface 38, the image-processing circuitproceeds to the step S101. If it is determined at the step S104 that thewriting implement has contacted the recording surface 38, theimage-processing circuit 17 obtains the distance “h” between theimage-formed locations of the contact point A and the optical axis 18 ofthe wide-angle lens 15 on the CMOS image sensor 16 at a step S105. InFIG. 11, the image-formed location of the optical axis 18 of thewide-angle lens 15 is set as a distance “h0” from the bottom left cornerof the CMOS image sensor 16. Additionally, the image-formed location ofthe contact point A is set as a distance “h1” from the bottom leftcorner of the CMOS image sensor 16. Then, the distance “h” can beobtained from the following equation.h=h0−h1

It should be noted that the distances “h0” and “h1” can be obtained fromthe number of pixels counted from the vertical side 39 to theimage-formed locations of the optical axis 18 and the contact point A,and a distance between pixels adjacent to each other (a pixel pitch).

Once a value of the distance “h” has been calculated, a value of theangle β1 can be obtained from the equations EQ1 and EQ2 by use ofpredetermined values of the distance “f” and the angle α at a step S106.The angle β2 can be obtained similarly by taking the above-describedsteps in the right camera unit 3. Additionally, at a step S107, thecoordinates (x, y) of the contact point A on the recording surface canbe obtained from the equations EQ3 and EQ4 by use of the values of theangles β1 and β2, and a predetermined value of the distance L. The CPU23 may execute a calculation using the equations EQ1 through EQ4.Alternatively, the image-processing circuits 17 and 22 may execute acalculation using the equations EQ1 and EQ2. Additionally, the CPU 23may execute a calculation using the equations EQ3 and EQ4.

The CPU 23 creates depiction data, for instance, by connecting each setof coordinates by straight lines based on a series of coordinate data ofthe contact point A that was obtained while the writing implement 36 wascontacting the recording surface 38. Subsequently, the CPU 23 displaysthe depiction data on the LCD 8 through the LCD display-control unit 28at a step S108, and stores the series of the coordinate data of thecontact point A in the flash memory 26, for instance, as a single file,at a step S109.

A description will now be given of a second embodiment of the presentinvention with reference to FIG. 12. The first embodiment describes acase in which the outline of the object is extracted for each frame ofthe image data after the object has been defined as a writing implementbased on the outline of the object. Instead, in the second embodiment,movement of the outline is monitored after the object has been definedas the writing implement, thereby enabling accurate detection of contactof the object with the recording surface 38. In the second embodiment,an optical flow measurement is applied to a method of detecting themovement of the outline. The optical flow measurement is to measure avelocity vector of each point of an image by using a rate of change in avalue of each pixel in a certain period and a rate of change in itssurrounding pixels in space.

FIG. 12 is a flowchart describing steps performed by the writing inputdevice 1 according to the second embodiment of the present invention. InFIG. 12, steps S201, S202 and S203 respectively correspond to the stepsS101, S102 and S103 shown in FIG. 8. At a step S204, each of theimage-processing circuits 17 and 22 obtains a velocity vector of eachpoint on the outline of the object that has been detected as a writingimplement by use of continuous frames of the image data of the writingimplement. At a step S205, each of the image-processing circuits 17 and22 checks whether a vector component of the velocity vector in adirection perpendicular to the recording surface 38 has become zero in acase that the writing implement tends to move toward the recordingsurface 38. If it is determined at the step S205 that the vectorcomponent has not become zero, the image-processing circuits 17 and 22proceed to the step S204. If it is determined at the step S205 that thevector component has become zero, each of the image-processing circuits17 and 22 checks whether the writing implement has contacted therecording surface 38 at a step S206. If it is determined at the stepS206 that the writing implement has not contacted the recording surface38, the image-processing circuits 17 and 22 proceed to the step S204. Ifit is determined at the step S207 that the writing implement hascontacted the recording surface 38, the image-processing circuits 17 and22 proceed to a step S207. Steps S207 through S211 correspond to thesteps S205 through S209, and thus a description of the steps S207through S211 will be omitted.

According to the present invention, objects such as a pencil, a stickand a finger may be used as writing implements if they are recognized aswriting implements by the writing input device 1. However, the penciland the stick that can be writing implements have different shapes.Accordingly, in a third embodiment of the present invention, typicalshapes of writing implements that are different from each other areinitially registered as data in the writing input device 1 so that auser can select one of the typical shapes. If the user selects one ofthe typical shapes appropriate for a writing implement that is to beused for recording information on the recording surface 38, an area ofan imaging device including pixels that output signals generated byphotoelectric conversion is changed depending on a selected shape.Accordingly, a load on the writing input device 1 to create image datais reduced.

FIG. 13 is a flowchart describing steps performed by the writing inputdevice 1 according to the third embodiment of the present invention.Data regarding shapes of outlines of typical writing implements shouldbe initially stored in the ROM 24 or in the flash memory 26. The userselects a writing-implement selecting mode by use of a selection keyprovided in the operation unit 27. At a step S301, the CPU 23 checkswhether an operation mode has become the writing-implement selectingmode. If the operation mode has become the writing-implement selectingmode, the CPU 23 displays icons corresponding to the typical writingimplements on the LCD 8 at a step S302. FIGS. 14A, 14B and 14C showexamples of the icons. FIG. 14A shows a first icon indicating writingimplements such as a ballpoint pen, a pencil and a mechanical pencilwhose holding parts are comparatively thin and whose tip is sharp. FIG.14B shows a second icon indicating writing implements such as a stickand a marking pen with a cap on. Additionally, FIG. 14C shows a third,icon indicating writing implements such as a finger. A height from therecording surface 38 in the photographing area 37 necessary for decidingshapes of the writing implements such as the pencil and the marking pencorresponding to the first and the second icons is less than that fordeciding the writing implements such as the finger corresponding to thethird icon while the writing implements corresponding to the first,second and third icons are contacting the recording surface 38.

Therefore, an area on the CMOS image sensors 16 and 21 including pixelsthat output signals generated by the photoelectric conversion is changeddepending to an icon selected by the user. At a step S303, the CPU 23checks whether the first icon or the second icon has been selected-bythe user. If it is determined at the step S303 that the first icon orthe second icon has been selected, the image-processing circuits 17 and22 reduce the number of pixels in the direction perpendicular to therecording surface 38 that output signals generated by the photoelectricconversion at a step S304, followed by proceeding to the step S101 ofthe first embodiment (FIG. 8). Subsequently, the steps S101 through S109are executed. If it is determined at the step S303 that the first iconor the second icon has not been selected, the CPU 23 checks whether thethird icon has been selected by the user, at a step S305. If it isdetermined at the step S305 that the third icon has been selected, theimage-processing circuits 17 and 22 increase the number of pixels in thedirection perpendicular to the recording surface 38 that output signalsgenerated by the photoelectric conversion at a step S306, followed byproceeding to the step S101 of the first embodiment (FIG. 8).Subsequently, the steps S101 through S109 are executed. It should benoted that the image-processing circuits 17 and 22 output a signalspecifying the area including pixels that output signals generated byphotoelectric conversion to the CMOS image sensors 16 and 21respectively.

A description will now be given of a fourth embodiment of the presentinvention with reference to FIG. 15. In the fourth embodiment of thepresent invention, typical shapes of writing implements that aredifferent from each other are initially registered as data in thewriting input device 1 so that a user can select one of the typicalshapes. If the user selects one of the typical shapes appropriate for awriting implement that is to be used for recording information on therecording surface 38, data about a selected shape is used as referencedata for deciding whether an object contacting the recording surface 38is for recording information on the recording surface 38 or not, therebyincreasing accuracy of the above-described decision process, as well asreducing the load on the writing input device 1 to create image data.

FIG. 15 is a flowchart describing steps performed by the writing inputdevice 1 according to the fourth embodiment of the present invention.Data related to shapes of outlines of typical writing implements shouldbe initially stored in the ROM 24 or in the flash memory 26, each datacorresponding to identification information of an icon. The typicalwriting implements correspond to the icons described in the thirdembodiment (FIG. 14). At a step S401, the CPU 23 checks whether anoperation mode has become the writing-implement selecting mode by a keyoperation executed by the user. If the operation mode has become thewriting-implement selecting mode, the CPU 23 displays the iconscorresponding to the typical writing implements on the LCD 8 at a stepS402. When detecting that one of the icons has been selected at a stepS403, the CPU 23 stores identification information of a selected icon inthe main memory 25. At a step S404, an information-inputting mode startsfollowing the writing-implement selecting mode. At a step S405, theimage-processing circuits 17 and 22 convert analog image signalsoutputted respectively from the CMOS image sensors 16 and 21 to digitalimage signals in order to obtain a frame of image data, and extract anoutline of the object from the frame of the image data. After theoutline of the object has been extracted, at a step S406, theimage-processing circuits 17 and 22 decide whether the object is awriting implement based on a shape of the extracted outline by using thedata related to shapes of outlines of typical writing implements thathas been stored in the ROM 24 or in the flash memory 26. In other words,the image-processing circuits 17 and 22 compare data related to theshape of the extracted outline with the data related to shapes ofoutlines of typical writing implements that corresponds to theidentification information of the selected icon stored in the mainmemory 25. If it is determined at the step S406 that both of the dataare identical, the image-processing circuits 17 and 22 determine theobject as a writing implement (step S103 in FIG. 8). Subsequently, thesteps S104 through S109 are executed.

A description will now be given of a fifth embodiment of the presentinvention with reference to FIG. 16. In a case that there is a patternsimilar to a writing implement in images of the writing implementphotographed by the left camera unit 2 and the right camera unit 3, thewriting input device might misunderstand the pattern as the writingimplement. In the fifth embodiment, the writing implement isphotographed to obtain data related to a shape of the writing implementbefore a user starts recording information by use of the writingimplement on the recording surface 38, and then the data is stored in amemory. The data is used as reference data for deciding whether anobject is the writing implement when the object is contacting therecording surface 38, thereby increasing accuracy of the above-describeddecision process, as well as reducing the load on the writing inputdevice 1 to create image data.

FIG. 16 is a flowchart describing steps performed by the writing inputdevice 1 according to the fifth embodiment of the present invention.Before the information-inputting mode, a writing implement isphotographed under a condition that the writing implement is contactingthe recording surface 38 or is close to the recording surface 38.Additionally, a shape-memory key provided in the operation unit 27 ispressed down by a user for storing data related to a shape of thewriting implement in the main memory 25. When the CPU 23 detects thatthe shape-memory key has been pressed down at a step S501, theimage-processing circuits 17 and 22 convert analog image signalsoutputted from the CMOS image sensors 16 and 21 to digital image signalsin order to obtain a frame of image data of the writing implement, andextract an outline of the writing implement from the obtained frame ofthe image data at a step S502. At a step S503, after the outline of thewriting implement has been extracted, the CPU 23 stores data related toa shape of the outline in the main memory 25, the data being used fordeciding the shape of the writing implement. Once the shape of theoutline is stored in the main memory 25, an operation mode shifts to theinformation-inputting mode at a step S504. At a step S505, theimage-processing circuits 17 and 22 convert analog image signalsoutputted from the CMOS image sensors 16 and 21 to digital image signalsin order to obtain a frame of image data of an object, and extract anoutline of the object from the obtained frame of the image data. At astep S506, after the outline of the object has been extracted, theimage-processing circuits 17 and 22 decide whether the objectcorresponds to the writing implement that was initially photographed,based on the outline of the object by use of the data that is related tothe shape of the writing implement and is stored in the main memory 25.In other words, the image-processing circuits 17 and 22 compare datarelated to the extracted outline of the object with the data related tothe shape of the writing implement. Subsequently, the image-processingcircuits 17 and 22 proceed to the step S103 shown in FIG. 8.

In the above-described embodiments, the writing input device 1recognizes only one object for recording information on the recordingsurface 38. However, there is a case that more than one object, forexample, a pen and a finger are recognized simultaneously as writingimplements by an image recognition method of the writing input device 1.A sixth embodiment provides a solution to the above-described case. Tobe concrete, when a plurality of objects has been recognizedsimultaneously as writing implements by the writing input device 1, thewriting input device 1 defines an object that is the closest to the leftcamera unit 2 and the right camera unit 3 as a writing implement. In thesixth embodiment, a description will be given of a case that there aretwo objects recognized simultaneously as writing implements. As shown inFIG. 17, paper is used as a recorded material, whereon a user writesinformation by use of a ballpoint pen with his or her right hand, andholds the paper with fingers of his or her left hand. In FIG. 17, acontact point A is a contact point of the ballpoint pen and the paper.Contact points B, C, D, E and F are contact points of the fingers andthe paper. It should be noted that the user's right hand is not shown inFIG. 17 since the right hand is not recognized as a writing implement.

FIG. 18 is a flowchart describing steps performed by the writing inputdevice 1 according to the sixth embodiment of the present invention. Atthe left camera unit 2, the image-processing circuit 17 converts ananalog image signal outputted from the CMOS image sensor 16 to a digitalimage signal in order to obtain a frame of image data of an object, andthen extracts an outline of the object from the frame of image data at astep S601. At a step S602, the image-processing circuit 17 compares theextracted outline with data related to shapes of writing implements suchas a pen and a finger that has been stored in the ROM 24 or in the flashmemory 26 in order to decide whether the object consists of a pluralityof writing implements. Since the thumb and the index finger of the lefthand are behind the middle finger of the left hand as shown in FIG. 17,the thumb and the index finger are not recognized as writing implements.Consequently, the other three fingers of the left hand and the ballpointpen are recognized as writing implements by the image-processing circuit17. Meanwhile, the above-described image recognition process isperformed at the right camera unit 3. At the right camera unit 3, theimage-processing circuit 22 converts an analog image signal outputtedfrom the CMOS image sensor 21 to a digital image signal in order toobtain a frame of image data of the object, and then extracts an outlineof the object from the frame of image data at the step S601. At the stepS602, the image-processing circuit 22 compares the extracted outlinewith the data related to shapes of writing implements that has beenstored in the ROM 24 or in the flash memory 26 in order to decidewhether the object consists of a plurality of writing implements. Sincethe ring finger and the little finger of the left hand are behind themiddle finger of the left hand as shown in FIG. 17, the ring finger andthe little finger are not recognized as writing implements.Additionally, the thumb and the index finger of the left hand are behindthe right hand that is not shown in FIG. 17, and thus the thumb and theindex finger are not recognized as writing implements. Consequently, themiddle finger of the left hand and the ballpoint pen are recognized aswriting implements by the image-processing circuit 22. Accordingly,writing implements that are recognized by both of the image-processingcircuits are the middle finger of the left hand and the ballpoint pen.

If it is determined by both of the image-processing circuits 17 and 22at the step S602 that the object consists of a plurality of the writingimplements as described above, the image-processing circuits 17 and 22decide whether the writing implements are contacting the recordingsurface at a step S603 similarly to the step S104 of the firstembodiment. If it is determined at the step S603 that the writingimplements are contacting the recording surface, the image-processingcircuits 17 and 22 calculate coordinates of the contact point A of theballpoint pen and the contact point D of the middle finger from theequations EQ1 through EQ4. In FIG. 19, the origin of coordinates isprovided as an origin P (0,0) at the top left corner of the recordingsurface. Additionally, a point Q (Xmax, 0) is given at the top rightcorner of the recording surface. A distance L1 is a distance from eachof the contact points A and D to the origin P. A distance L2 is adistance from each of the contact points A and D to the point Q.Assuming the coordinates of the contact point A are (x, y), thedistances L1 and L2 for the contact point A are obtained by use of thefollowing equations.L1=√(x ² +y ²)L2=√((Xmax-−x)² +y ²)

Similarly, the distances L1 and L2 for the contact point D are obtained.Subsequently, at a step S604, the sum of the distances L1 and L2 iscalculated for each of the contact points A and D. In the sixthembodiment, since the sum for the contact point A is greater than thatfor the contact point D, the image-processing circuits 17 and 22 definean object contacting the paper at the contact point A, that is, theballpoint pen as a valid writing implement. At a step S605, in the casethat the image-processing circuits 17 and 22 recognize a plurality ofwriting implements contacting the recording surface, theimage-processing circuits 17 and 22 obtain coordinates of a contactpoint where the sum of the distances L1 and L2 is the smallest, as validcoordinate data, The steps S108 and S109 are executed after the stepS605.

If it is determined by both of the image-processing circuits 17 and 22at the step S602 that the object does not consist of a plurality ofwriting implements, the image-processing circuits 17 and 22 proceed to astep S606, and check whether a single writing implement has beenrecognized. If not, the image-processing circuits 17 and 22 proceed tothe step S601. If it is determined at the step S606 that a singlewriting implement has been recognized, the image-processing circuits 17and 22 obtain coordinates of a contact point of the single writingimplement. Subsequently, the steps S108 and S109 of the first embodimentare executed.

If it is determined at the step S603 that a plurality of writingimplements are not contacting the recording surface at a step S603, theimage-processing circuits 17 and 22 proceed to the step S608, and checkwhether a single writing implement is contacting the recording surface.If not, the image-processing circuits 17 and 22 proceed to the stepS601. If it is determined at the step S608 that a single writingimplement is contacting the recording surface, the image-processingcircuits 17 and 22 obtain coordinates of a contact point of the singlewriting implement. Subsequently, the steps S108 and S109 of the firstembodiment are executed.

In FIG. 19, an area near a line connecting the origin P and the point Qincludes an area that cannot be detected as a recording area of thewriting input device 1. However, such a problem can be solved bychanging specifications of the writing input device 1. For instance, theorigin P and the point Q may be provided at locations that are certaindistance away in a Y-direction from the writing input device instead oflocations contacting the writing input device 1.

According to the sixth embodiment, in a case that a plurality of writingimplements such as a pen and a finger is recognized by camera units, anobject that is the closest to the camera units is selected as the onlywriting implement. For example, while a user is writing information on arecording sheet with a pen as well as holding the recording sheet by hisor her hand, written data of the pen is recorded in a recording mediumas electric data if the pen is placed closer than fingers to the cameraunits. Additionally, the writing input device 1 according to the sixthembodiment can prevent a user from inputting undesired information tothe writing input device 1 in a case that an object other than writingimplements is recognized as a writing implement by mistake.

A seventh embodiment of the present invention enables simple managementof data inputted by a writing implement by defining a size of arecording area as a standard paper size. FIG. 20 is a flowchartdescribing steps performed by the writing input device according to theseventh embodiment of the present invention. A length of the recordingarea 13 in the vertical direction shown in FIG. 4 is defined, forinstance, as a height of a letter size (A4). Such information isinitially recorded in the ROM 24. At a step S701, coordinates of acontact point of a writing implement contacting the recording surfaceare obtained by use of one of the methods described in the first throughfifth embodiments. If the obtained coordinates of the contact point arein the defined recording area at a step S702, the CPU 23 processes datarelated to the obtained coordinates as valid data at a step S703. To beconcrete, the CPU 23 creates depiction data based on a series ofcoordinate data of the contact point. Subsequently, the CPU 23 displaysthe depiction data on the LCD 8 through the LCD display-control unit 28and stores the series of the coordinate data of the contact point in theflash memory 26. On the other hand, if the obtained coordinates of thecontact point are located outside the defined recording area at the stepS702, the CPU 23 defines the data related to the obtained coordinates asinvalid data, and does not execute the above-described displaying andstoring processes at a step S704.

In an eighth embodiment of the present invention, a user can set a sizeof the recording area 13 to one of a letter size (A4) with a longer sideplaced in the vertical direction (an A4 height), the letter size with ashorter side placed in the vertical direction (an A4 width), a legalsize (B4) with a longer side placed in the vertical direction (a B4height), the legal size with a shorter side placed in the verticaldirection (a B4 width), and the like. Such information is initiallystored in the ROM 24. Additionally, width of the recording area 13 canbe altered as a distance changes between the left camera unit 2 and theright camera unit 3 in the eighth embodiment. The expansion/contractionunits 6 and 7 shown in FIG. 2 can expand and contact by a handoperation. By use of the expansion/contraction units 6 and 7, a user canset the width of the recording area 13 to any of the A4 height, the A4width, the B4 height and the B4 width. When both of theexpansion/contraction units 6 and 7 contract the most, the distancebetween the left camera unit 2 and the right camera unit 3 becomes theshortest. Consequently, a size of the recording area 13 is set to the A4height. When both of the expansion/contraction units 6 and 7 expand themost, the distance between the left camera unit 2 and the right cameraunit 3 becomes the longest. Consequently, a size of the recording area13 is set to the B4 width. Additionally, when the expansion/contractionunit 6 on the left contracts the most, and the expansion/contractionunit 7 on the right expands the most, a size of the recording area 13 isset to the B4 height. Additionally, when the expansion/contraction unit6 on the left expands the most, and the expansion/contraction unit 7 onthe right contracts the most, a size of the recording area 13 is set tothe A4 width. Each of the expansion/contraction units 6 and 7 isprovided with a switch that is not shown in the figures, and thus astate of expansion/contraction of each of the expansion/contractionunits 6 and 7 can be detected by checking the switch. In other words, bychecking a combination of on/off settings of switches provided in theexpansion/contraction units 6 and 7, the CPU 23 can detect which papersize is selected by a user.

FIG. 21 is a flowchart describing steps performed by the writing inputdevice according to the eighth embodiment of the present invention. At astep S801, the CPU 23 checks a state of expansion/contraction of each ofthe expansion/contraction units 6 and 7. Subsequently, the CPU 23selects a size of the recording area 13 corresponding to the state ofexpansion/contraction of each of the expansion/contraction units 6 and 7from the above-described four sizes at a step S802. For instance, whenboth of the expansion/contraction units 6 and 7 have contracted themost, the CPU 23 sets a size of the recording area 13 to the A4 height.At a step S803, the image-processing circuits 17 and 22 obtaincoordinates of a contact point of a writing implement contacting therecording surface by performing the steps described in one of the firstthrough fifth embodiments. If the obtained coordinates of the contactpoint are in the selected recording area at a step S804, the CPU 23processes data related to the obtained coordinates as valid data at astep S805 similarly to the step S703 shown in FIG. 20. On the otherhand, if the obtained coordinates of the contact point are locatedoutside the selected recording area at the step S804, the CPU 23 definesthe data related to the obtained coordinates as invalid data, and doesnot process the data at a step S806. It should be noted that four typesof the states of expansion/contraction are provided in the writing inputdevice 1 for setting the size of the recording area 13. However, thenumber of the states can be increased so that there can be moreselections for selecting the size of the recording area 13.

According to the eighth embodiment, since a desired recording area canbe selected from a plurality of recording areas whose shapes and sizesare different from each other, operability of the writing input device 1increases.

A description will now be given of a ninth embodiment of the presentinvention with reference to FIG. 22. According to the ninth embodiment,before recording information in the recording area 13, a user can set asize of the recording area 13 freely in a range where coordinates of awriting implement can be obtained in addition to predetermined sizes ofthe recording area 13. In other words, the user can set an area largerthan the predetermined sizes of the recording area 13 as the recordingarea 13.

FIG. 22 is a flowchart describing steps performed by the writing inputdevice according to the ninth embodiment of the present invention. Whena recording-area setting mode is set at a step S901 by a user operatingkeys provided in the operation unit 27, the image-processing circuits 17and 22 process a track of coordinates of a contact point of a writingimplement on the recording surface as information used for setting thesize of the recording area 13 instead of processing the track as datarelated to an outline of the writing implement, at a step S902. In therecording-area setting mode, the image-processing circuits 17 and 22decide whether the track of the coordinates of the contact point forms aclosed area. The closed area may be an area whose outline is formed byonly the track of the coordinates of the contact point. Alternatively,the closed area may be an area whose outline is formed by a straightline PQ connecting the origin P and the point Q shown in FIG. 19 and thetrack of the coordinates of the contact point. Furthermore, the closedarea may be an area whose outline is formed by straight lines PQ, PR andQS shown in FIG. 23 and the track of the coordinates of the contactpoint. It should be noted that the straight line PR is drawnperpendicularly to the straight line PQ from the origin P. The straightline QS is drawn perpendicularly to the straight line PQ from the pointQ. When a user presses down a confirmation key provided in the operationunit 27 at a step S903 after specifying an area by moving the writingimplement on the recording surface, the image-processing circuits 17 and22 define the area as the recording area 13 at a step S904.

For instance at the step S901, by use of the operation unit 27 and aguidance displayed on the LCD 8, a user sets one of recording-areasetting modes wherein the straight lines PQ, PR and QS are fixed so thata user can specify only one side RS of a rectangle PQSR as shown in FIG.23. Subsequently, when a user has pressed down a confirmation keyprovided in the operation unit 27 at the step S903 after drawing theside RS by use of the writing implement, the image-processing circuits17 and 22 define the rectangle PQSR as the recording area 13 at the stepS904, and store coordinates of each of the points P, Q, Rand S asrecording-area information in the main memory 25.

After the recording area has been defined at the step S904, therecording-area setting mode is cancelled, and the information-inputtingmode is set at a step S905. Steps S906 through S909 correspond to thesteps S701 through S704 respectively.

If it is determined that the confirmation key has not been pressed atthe step S903, the image-processing circuits 17 and 22 proceed to thestep S902. If the recording area 13 could not been formed at the stepS904, the CPU 23 displays a message on the LCD 8 at a step S910 tonotify a user about failure of the formation of the recording area 13,and proceeds to the step S902.

In the above-described ninth embodiment, the recording area 13 is therectangle PQSR. However, the writing input device 1 can manageinformation about the recording area 13 even if the shape of therecording area 13 is a shape other than a rectangle.

According to a tenth embodiment, when the writing input device 1 detectsthat a user has written information outside the recording area 13, thewriting input device 1 notifies a user that an invalid writing operationhas executed by sounding an alarm. FIG. 24 is a flowchart describingsteps performed by the writing input device according to the tenthembodiment of the present invention. At a step S1001, theimage-processing circuits 17 and 22 obtain coordinates of a contactpoint of an object that has been recognized as a writing implement inthe seventh, eighth and ninth embodiments. Subsequently, at a stepS1002, the image-processing circuits 17 and 22 determine whether thecontact point of the writing implement is in the recording area 13. Ifit is determined at the step S1002 that the contact point is in therecording area 13, the writing input device 1 processes data related tothe obtained coordinates as valid data at a step S1003. If it isdetermined at the step S1002 that the contact point is located outsidethe recording area 13, the writing input device 1 does not processcoordinate data of the contact point at a step S1004. Additionally, at astep S1005, the writing input device 1 outputs an alarm signal from thesound source 30 to the speaker 32 wherefrom an alarming sound isoutputted. According to the tenth embodiment, a user can easily identifythe recording area 13 even in a case that a recording sheet such aspaper is not used as the recording area 13. In other words, even in acase that the recording area 13 is hardly determined because of arecording surface being a desk or a wall, a user can write informationin the recording area 13 smoothly and correctly.

In an eleventh embodiment, a frame is provided on the edge of therecording area 13 so that a user can easily notice a range of therecording area 13. Additionally, from an image photographed by the leftcamera unit 2 and the right camera unit 3, the image-processing circuits17 and 22 do not detect a contact point of a writing implement on arecorded material such as paper in an area outside the recording area 13where the contact point is behind the frame, thereby reducing the loadon the writing input device 1 to execute image processes.

In a case that the image-processing circuits 17 and 22 extract anoutline of an object from an image photographed by the left camera unit2 and the right camera unit 3, a larger contrast of the object and itssurrounding area produces a higher accuracy in extracting the outline ofthe object. In other words, a larger difference in luminance of theobject and its surrounding area produces a higher accuracy in extractingthe outline of the object. If a writing implement is specified, abackground color of the image photographed by the left camera unit 2 andthe right camera unit 3 can be set to a color which is the mostappropriate to a color of the writing implement. However, if the colorof the writing implement is not specified, the background color of theimage should be set to a color by which objects with various colors canbe easily extracted from the image. For instance, it is assumed that afinger is used as a writing implement. In such case, when the finger isphotographed under a regular room light, luminance of the finger iscloser to luminance of a white subject than to that of a black subjecthaving the lowest luminance. Accordingly, in the eleventh embodiment, acolor of the inner surface of the frame provided on the edge of therecording area 13 is set to black, thereby increasing accuracy inrecognizing a shape of an unspecified writing implement and detectingcoordinates of a contact point of the unspecified writing implement on arecorded material.

In a case that a sheet of paper and the like are not used as therecording area 13, a frame is provided on the edge of the recording area13 so that a user can easily distinguish the recording area 13. As shownin FIG. 25, a frame made of a plastic and the like includes three bars43, 44 and 45. The three bars 43, 44 and 45 can be combined or separatedby use of combination units 49 and 50. Shapes of the combination units49 and 50 are existing shapes. A height of the frame from the recordingsurface should be set high enough not to be able to recognize a state inwhich an object recognized as a writing implement is contacting therecording surface outside the recording area 13. By providing such aframe on the edge of the recording area 13, a recognition process of acontact point of the object on the recording surface outside therecording area 13 cannot be executed, thereby sharply reducing an amountof processes performed by the writing input device 1. Additionally, bysetting a color of inner surfaces 46, 47 and 48 of the above-describedbars 43, 44 and 45 to black, and further by using materials having lowlight-reflectance as materials used for creating the bars 43, 44 and 45,accuracy in recognizing a shape of an unspecified writing implement anddetecting coordinates of a contact point of the unspecified writingimplement on the recording surface is increased.

A description will now be given of a twelfth embodiment of the presentinvention. An amount of electric charge stored in each pixel on a CMOSimage sensor in a unit time by photoelectric conversion depends on anamount of light irradiated onto an imaging device. In other words, theamount of electric charge stored in each pixel by the photoelectricconversion in a unit time increases as an amount of incident light tothe imaging device increases, and thus a frame rate outputted as imagesignals can be increased. In addition, a user writes information on arecorded material occasionally in a place where a lighting environmentis insufficient. Accordingly, in the twelfth embodiment, accuracy inrecognizing a shape of a writing implement and detecting a contact pointof the writing implement on the recorded material is increased byirradiating light against the recording area 13 located on the recordedmaterial. When the writing input device 1 is powered on, the LED 9, theLED 10, the LED 11, and the LED 12 are supplied with electric current,and turned on. Consequently, the amount of incident light irradiated tothe CMOS image sensors 16 and 21 increases. Thus, the amount of electriccharge stored in each pixel by the photoelectric conversion in a unittime increases so that a frame rate outputted as image signals can beincreased.

Accordingly, accuracy in recognizing a shape of an unspecified writingimplement and detecting coordinates of a contact point of theunspecified writing implement on a recording surface can be increased.The twelfth embodiment becomes very effective especially in a case thatthe writing input device 1 is used in a place where the lightingenvironment is insufficient. The description has been given of thetwelfth embodiment in which each LED is turned on when the writing inputdevice 1 is powered on. Alternatively, each LED in the writing inputdevice 1 may be supplied with a switch, and may be turned on when aswitch corresponding the LED is pressed down.

In a thirteenth embodiment, in a case that a resolution of readingcoordinates is low, by setting an entire reading area (the recordingarea 13) as a part of a page of a data area, written data such asletters inputted by use of a writing implement is combined with a pageof written data, and is displayed on an image-display unit, or is storedas a file. Additionally, in a case that a small image-display devicethat can only display a small number of pixels is used as animage-display device for displaying the written data, the thirteenthembodiment enables displaying images on the image-display device bydividing a page of the data area with a large number of pixels, andwriting information in the divided data area. In other words, since apage of the data area is divided into a plurality of blocks, wherein oneof the blocks is assigned to a recording area, a page of written datacan be created by inputting information in the recording area a numberof times even if a resolution of reading information written in therecording area is low.

A range of the recording area is limited to an area where a writingimplement can be recognized, as shown in FIG. 4. However, in thethirteenth embodiment, a description will be given of a case that therecording area is set to a letter (A4) size including the points P, Q, Rand S as corners of the recording area as shown in FIG. 23. The point Pis the origin of coordinates of contact points of a writing implement inthe recording area. The point Q is placed at the end of an X-axis drawnfrom the origin P. A distance between the points P and Q is set to ashorter side (width) of the A4 size by use of the extraction/contractionunits 6 and 7 shown in FIG. 2 by a hand operation. In addition, thedistance between the points P and Q is displayed in millimeters on theLCD 8. If the distance between the points P and Q is equal to a width ofa fixed sheet size, the LCD 8 displays letters such as “A4 width” and“B4 width”. Accordingly, a user can check a width of the recording areawhile writing information in the recording area since a distance betweenthe left camera unit 2 and the right camera unit 3 is displayed inmillimeters or by a sheet size on the LCD 8.

A description will now be given of a method of managing written data.The written data is managed by a page having a fixed size. A recordingarea can be set to any size so that a size of the page can be set to anyvalue. In the thirteenth embodiment, it is assumed that a data size (thenumber of pixels) of the page is set to an A4 data size (the number ofpixels). The number of pixels included in the page is set to an864-pixel width by an 1140-pixel length based on an ITU-T T.4 related toa document of a group-3 facsimile device. The above-described size ofthe page is equivalent to 100 dpi (dot per inch) in the width and thelength of the page. A page of written data or a plurality of pages ofwritten data may be stored as one file in the flash memory 26.

The description has been given of the method of obtaining coordinates ofcontact points or writing positions of a writing implement by using theequations EQ1 through EQ4 when units are placed in positions' shown inFIG. 6. In the method, the only variable parameter used for obtainingthe coordinates is the angle θ shown in the equation EQ1, or thedistance “h” between an image-formed location of the optical axis of awide-angle lens and an image-formed location of a writing position on aCMOS image sensor.

In the above-described method, a resolution of reading informationwritten at the writing position depends on a distance measured from thewriting position to the left camera unit 2 or to the right camera unit3. A concrete description will be given of a relation between thedistance and the resolution with reference to FIG. 26. FIG. 26 showspositions of a CMOS image sensor, a wide-angle lens, writing positions(writing points) A, B, C and D. It is assumed that the points A and Care placed on the optical axis of the wide-angle lens for a descriptionpurpose. A point E is an image-formed location of the points A and C,and is on a surface of the CMOS image sensor. A point F is animage-formed location positioned at a pixel adjacent to a pixelcorresponding to the point E. The points B and D are placed at certainpoints so that an image of a subject placed at the point B or D can beformed at the point F. It is assumed that angles between lines AB andAE, and between lines CD and CE are 90 degrees.

Additionally, a cross section of the line AE and a line BF is named apoint G. By setting an angle between lines AG and BG, or between CG andDG as an angle θ, a length of the line AG as a length d1, a length ofthe line CG as a length d2, a length of the line AB as a length λ1, anda length of the line CD as a length λ2, the lengths λ1 and λ2 areobtained by the following equations.λ1=d1*tan θ  EQ5λ2=d2*tan θ  EQ6

It is obvious from the above equations EQ5 and EQ6 that distance rangesλ1 and λ2 of a subject photographed by pixels E and F (pixels where thepoints E and D are located), which are adjacent to each other on thesurface of the CMOS image sensor, vary depending on the distances d1 andd2 from the wide-angle lens. The pixels E and F photograph the subjectlocated at the distance d1 from the point G in the range λ1. On theother hand, the pixels E and F photograph the subject located at thedistance d2 from the point G in the range λ2. This indicates thataccuracies in reading coordinates of the writing position are the samein a case of writing information from the point A to the point B, and ina case of writing information from the point C to the point D.

A resolution measured in dpi of reading the coordinates of the writingposition on the line AB is obtained by dividing one inch that is a unitlength by the distance λ1 (inch). In addition, a resolution of readingthe coordinates of the writing position on the line CD is obtained bydividing one inch by the distance λ2 (inch). In a case that the numberof pixels that can be read by one of the left camera unit 2 and theright camera unit 3 in a horizontal direction of the CMOS image sensors16 and 21, that is, a direction parallel to the recording surface 38 ina photographed image shown in FIG. 9, is 640 pixels, a resolution ofreading coordinates at the points R and S is about 28 dpi in an A4-sizedrecording area shown in FIG. 23. Additionally, a resolution of readingcoordinates at the center of the A4-sized recording area is about 56dpi. As described above, the closer to the left camera unit 2 and theright camera unit 3 the writing position is in the recording area, themore densely the coordinates are read by the CMOS image sensors 16 and21. On the contrary, the farther from the left camera unit 2 and theright camera unit 3 the writing position is in the recording area, themore roughly the coordinates are read by the CMOS image sensors 16 and21.

Now, it is assumed that the LCD 8 can display 432 pixels in thehorizontal direction and 285 pixels in the vertical direction. In a caseof displaying a page of written data whose size is the 864-pixel widthby the 1140-pixel length on the LCD 8, the page must be divided intoseveral blocks. FIG. 27 is a block diagram showing an embodiment of amethod to divide the page of written data. As shown in FIG. 27, thenumber of pixels included in the page of written data is divided by thenumber of pixels displayed on the LCD 8 into eight displaying blocks 51.In other words, the number of pixels in each of the divided displayingblocks 51 is equal to the number of pixels displayed on the LCD 8. Thisdisplaying block 51 (432×285 pixels) is separated into a left block anda right block, each of the left and the right blocks having the size of216×285 pixels. Subsequently, each of the left and right blocks is nameda writing block 52, and is assigned to the A4-sized recording area shownin FIG. 23 to fit to the A4 size. FIG. 28 shows an embodiment in which apage of written data is divided into a plurality of the writing blocks(recording areas) 52. A size of the writing block 52 is exactly equal toa half of the displaying block 51 so that a page of written data isevenly divided into sixteen blocks. FIG. 29 shows a relation between apage of written data and the recording area in a case of writinginformation in a fourth writing block (a fourth recording area) shown inFIG. 28. A size of a recording area shown in FIG. 29 is a letter size(A4). In addition, the recording area is provided with paper whereon aline of letters is written by use of a pen.

A description will now be given of processes performed in the thirteenthembodiment in a case of writing information in one of the writing blocks52 created as described above. FIG. 30 is a flowchart describing stepsperformed by the writing input device according to the thirteenthembodiment of the present invention. Steps S1101 through S1107 areexecuted similarly to the steps S101 through S107 of the firstembodiment shown in FIG. 8 in order to obtain coordinates (x, y) of acontact point of a writing implement on a recording surface. A unit ofthe coordinates (x, y) is the same as a unit of the distance L shown inFIG. 6, and is millimeters in the thirteenth embodiment. Since the sizeof the writing block 52 is exactly 1/16 of the size of a page of writtendata, the coordinates (x, y) of the contact point (writing position) inthe recording area are expressed in the numbers of pixels counted fromthe origin of the page in an X-axis direction and in a Y-axis direction.Initially, assuming a resolution of reading coordinates (an interval ofreading coordinates) is even throughout the recording area (the writingblock 52), coordinates (x_dot, y_dot) expressed in pixels are calculatedfrom the coordinates (x, y) expressed in millimeters in one of thewriting blocks 52 by use of the following equations at a step S1108.x_dot=x/(an interval of reading coordinates in the X-axisdirection)  EQ7y_dot=y/(an interval of reading coordinates in the Y-axisdirection)  EQ8

Subsequently, by setting the top left corner of a page of written dataas the origin of the page, an origin of an “n”th writing block 52 can beexpressed as coordinates (x_org(n), y_org(n)). A range of the value “n”is from 0 to 15. At a step S1109, coordinates (x_dot_page, y_dot_page)of the writing position in a page of written data are obtained from thefollowing equations.x_dot_page=x_org(n)+x_dot  EQ9y_dot_page=y_org(n)+y_dot  EQ 10

The writing input device 1 manages the coordinates (x_dot_page,y_dot_page) expressed in pixels as written data. For instance, in a casethat information is written at coordinates (100, 200) in a tenth writingblock, coordinates (100, 200) in the tenth writing block are convertedinto coordinates (316, 770) in a page. It should be noted thatcoordinates of the origin of the tenth writing block are (216, 570).

At a step S1110, the CPU 23 stores a series of coordinate data of acontact point that has been obtained while the writing implement wascontacting the recording surface in one of the writing blocks in amemory as written data of the writing block in a page of written data.In addition, the CPU 23 creates depiction data from the series of thecoordinate data by use of a method of connecting each set of coordinateswith a straight line, for example, and displays the depiction data onthe LCD 8 through the LCD display-control unit 28 at a step S1111.

It is assumed in the thirteenth embodiment that the resolution ofreading coordinates is even throughout the entire recording area whenobtaining the coordinates (x_dot, y_dot) expressed in pixels in order tosimplify the description. However, in reality, the resolution of readingthe coordinates varies depending on a distance from the left camera unit2 or the right camera unit 3 to the coordinates (x_dot, y_dot). A methodof making a pixel density of written data even throughout the entirerecording area will be described later in other embodiments.

Additionally, the following equations EQ11 and EQ12 can be substitutedfor the equations EQ7 and EQ8 for obtaining the coordinates (x_dot,y_dot) expressed in pixels from the coordinates (x, y).x_dot=864×(x/215)×(¼)  EQ11y_dot=1140×(y/290)×(¼)  EQ12

The equations EQ11 and EQ12 are derived from the following facts. Thenumbers of pixels in the width and the length of a page of written dataare respectively 864 pixels and 1140 pixels. The width and the length ofthe page are respectively 215 millimeters and 290 millimeters. Inaddition, the width and the length of one of the writing blockscorresponding to the recording area are ¼ of the width and the length ofthe page respectively.

In a fourteenth embodiment of the present invention, the recording areais assigned to any area in a page of a data area. Additionally, in acase that the resolution of reading coordinates in the recording area islow, a page of written data is created by writing data in the recordingarea several times. In other words, in the above-described thirteenthembodiment, the description has been given of the method of dividing thenumber of pixels in a page of the data area into 8 blocks, displayingone of the blocks on the LCD 8, and assigning half of a block to therecording area. On the other hand, in the fourteenth embodiment, adescription will be given of a method of assigning a pixel-displayingarea of the LCD 8 to any area in a page of the data area, and thenassigning a part of the pixel-displaying area to the recording area.Similarly to the thirteenth embodiment, the LCD 8 can display 432 pixelsin the horizontal direction and 285 pixels in the vertical direction. Alocation of the pixel-displaying area of the LCD 8 can be moved freelyto any location in a page of the data area by a user operation. A methodof moving the pixel-displaying area of the LCD 8 will be described laterin other embodiments.

FIG. 31 is a block diagram showing an embodiment of a pixel-displayingarea 53 of the LCD 8 in a page of the data area. FIG. 31 furtherincludes a displaying area 54 for writing that is a 216 pixel by 285pixel shaded area located at the center of the pixel-displaying area 53.In other words, coordinates of the origin (the top left corner) of thepixel-displaying area 53 are (300, 300) in FIG. 31. Additionally,coordinates of the origin of the displaying area 54 for writing are(408, 300). In the same manner as the thirteenth embodiment, thecoordinates (x, y) of a contact point (a writing position) of a writingimplement contacting a recording surface are obtained. Subsequently, thecoordinates (x, y) are converted to the coordinates (x_dot, y_dot)expressed in pixels by use of the equations EQ7 and EQ8. Assuming thatthe top left corner of a page of the data area is the origin,coordinates of the starting point of the recording area, that is, thetop left corner of the recording area, expressed in pixels are namedcoordinates (x_org, y_org). Coordinates (x_dot_page, y_dot_page) of thewriting position in a page of the data area are expressed as below.x_dot_page=x_org+x_dot  EQ13y_dot_page=y_org+y_dot  EQ14

For example, if data is written at coordinates (100, 200) in thedisplaying area 54 for writing shown in FIG. 31, the coordinates (100,200) in the displaying area 54 for writing are converted to coordinates(508, 500) in a page of the data area. The writing input device 1manages the coordinates (x_dot_page, y_dot_page) expressed in pixels aswritten data.

As described above, the fourteenth embodiment is characterized by thefunction to assign the pixel-displaying area 53 to any location in apage of the data area. After a location of the pixel-displaying area 53has been set in a page of the data area, the steps S1110 and S1111 shownFIG. 30 of the thirteenth embodiment are executed. It should be notedthat a displaying area for writing is substituted for a writing block inthe step S1110 in the fourteenth embodiment. According to the fourteenthembodiment, the recording area is assigned to a desired area in a pageof the data area so that the writing input device 1 can create andmanage a page of written data that includes a large number of pixels bywriting data in the recording area a plurality of times even in a casethat the resolution of reading coordinates is low in the recording area.Additionally, operability of the writing input device 1 increases when auser writes data in the desired area of a page of the data area.

In a fifteenth embodiment, an area corresponding to a recording area isdisplayed on a displaying unit distinctively so that a user canrecognize easily which part of a page of a data area he or she iscurrently writing data in, thereby improving operability of the writinginput device 1. In the above-described thirteenth embodiment, in a casethat a writing block corresponding to the recording area is displayed ona monitor of the LCD 8, the writing block is displayed in the right orleft half of a displaying block as shown in FIGS. 27 and 28. On theother hand, in the fourteenth embodiment, a displaying area for writingthat corresponds to the recording area is displayed at the center of themonitor of the LCD 8, as shown in FIG. 31. In the fifteenth embodiment,a border of the displaying area for writing is accentuated on themonitor of the LCD 8 in order for a user to easily distinguish thedisplaying area for writing. FIG. 32A shows an embodiment correspondingto the thirteenth embodiment of an emphasized displaying area 55 forwriting on the monitor of the LCD 8. FIG. 32B shows an embodimentcorresponding to the fourteenth embodiment of an emphasized displayingarea 56 for writing on the monitor of the LCD 8.

In a sixteenth embodiment, a description will be given of a method ofdisplaying an area corresponding to a recording area on a displayingunit distinctively so that a user can recognize easily which part of apage of a data area he or she is currently writing data in, therebyimproving operability of the writing input device 1. In the thirteenthembodiment, the size of a writing block is set to exactly 1/16 of thesize of a page of the data area as shown in FIG. 28. In such case, it isconvenient to recognize a location of a writing block corresponding to acurrent recording area by displaying the location of the writing blockwith locations of all the writing blocks. Accordingly, a block-locationdisplaying key is provided in the operation unit 27 for displaying thewriting block on the LCD 8.

FIG. 33 is a flowchart describing steps performed by the writing inputdevice according to the sixteenth embodiment of the present invention.At a step S1201, the writing input device 1 checks whether theblock-location displaying key has been pressed. If it is determined atthe step S1201 that the block-location displaying key has been pressed,a situation 57 in which a page is divided into blocks is displayed onthe LCD 8, wherein a location of a writing block 58 is specified. InFIG. 34, the location of the writing block 58 (a writing block 10) isdisplayed with its luminance being inverted from other blocks in a pageof the data area, at a step S1202. If it is determined at a step S1203that the block-location displaying key has been pressed again, the LCD 8displays image data that had been displayed previously before thesituation 57 was displayed thereon, for instance, a screen shown in FIG.32A or FIG. 32B, at a step S1204. According to the sixteenth embodimentof the present invention, an area corresponding to the recording area isdisplayed on the displaying unit distinctively so that a user canrecognize easily which part of a page of the data area he or she iscurrently writing data in, thereby improving operability of the writinginput device 1.

A description will now be given of a seventeenth embodiment of thepresent invention. In the seventeenth embodiment, writing data in anypart of a page of a data area is enabled since any of writing blockscorresponding to a recording area can be selected, thereby improvingoperability of the writing input device 1. FIG. 35 is a flowchartdescribing steps performed by the writing input device according to theseventeenth embodiment of the present invention. In the seventeenthembodiment, a direction-specifying key including up, down, left andright buttons is provided in the operation unit 27. If it is determinedat a step S1301 that the block-location displaying key has been pressed,a location of the writing block 58 (a writing block 10) is displayed onthe LCD 8 with its luminance being inverted from other blocks in a pageof the data area as shown in FIG. 34, at a step S1302. Subsequently, ifit is determined at a step S1303 that the direction-specifying key hasbeen pressed, the writing block 58 corresponding to the recording areais moved to a direction specified by use of the direction-specifyingkey, and is displayed with its luminance being inverted at a step S1304.For example, in a case that the writing block 10 shown in FIG. 34corresponds to the writing block 58, a writing block corresponding tothe recording area is set to a writing block 11 when the right button ofthe direction-specifying key is pressed once, and is displayed with itsluminance being inverted. FIG. 36 shows the LCD 8 displaying that awriting block 59 corresponding to the recording area is moved from thewriting block 10 to a writing block 11. Luminance of the writing block59 before having been moved to the writing block 11 is inverted back tothe original value at a step S1305.

According to the seventeenth embodiment, writing data in any part of apage of the data area is enabled since any of the writing blockscorresponding to the recording area can be selected, thereby improvingoperability of the writing input device 1.

An eighteenth embodiment enables free movement of an area correspondingto the recording area in a page of a data area, and thus simplifieswriting data in any part of a page of the data area, thereby improvingoperability of the writing input device 1. The direction-specifying keyis used for selecting a writing block in the seventeenth embodiment. Onthe other hand, the direction-specifying key is used for moving thepixel-displaying area 53 and the displaying area 54 for writing by apixel in the eighteenth embodiment. The pixel-displaying area 53 of theLCD 8 and the displaying area 54 for writing that corresponds to therecording area in the pixel-displaying area 53 are shown in FIG. 31.When the direction-specifying key is pressed, the pixel-displaying area53 or the displaying area 54 for writing is moved by a fixed number ofpixels to a direction specified by use of the direction-specifying key.The operation unit 27 additionally includes a moving-area selecting keyused for selecting either the pixel-displaying area 53 or the displayingarea 54 for writing as an object of moving. By use of the moving-areaselecting key, one of the pixel-displaying area 53 and the displayingarea 54 for writing is selected and is moved to the direction specifiedby use of the direction-specifying key.

FIG. 37 is a flowchart describing steps performed by the writing inputdevice according to the eighteenth embodiment of the present invention.A description will initially be given of a case of moving thepixel-displaying area 53 of the LCD 8. The LCD 8 can display an imagehaving a size of a 432-pixel width by a 285-pixel length at most. A userinitially selects one of the pixel-displaying area 53 and the displayingarea 54 for writing by use of the moving-area selecting key, followed bypressing the direction-specifying key to move the selected area. Thewriting input device 1 checks whether the moving-area selecting key hasbeen pressed at a step S1401. If the writing input device 1 determinesat the step S1401 that the moving-area selecting key has been pressed,the writing input device 1 proceeds to a step S1402, and checks whethera selected area is the pixel-displaying area 53 of the LCD 8. If it isdetermined at the step S1402 that the pixel-displaying area 53 has beenselected, the writing input device 1 further checks whether thedirection-specifying key has been pressed at a step S1403. If it isdetermined at the step S1403 that the direction-specifying key has beenpressed, a desired area (the pixel-displaying area 53) in a page of thedata area is displayed on the LCD 8 after being moved in a directionspecified by use of the direction-specifying key, at a step S1404.

FIG. 38 is a diagram showing an embodiment of a pixel-displaying area 60displayed on the LCD 8. In FIG. 38, a displaying area 61 for writingthat corresponds to the recording area is displayed at the center of thepixel-displaying area 60. The pixel-displaying area 60 displayed on theLCD 8 additionally includes a summarized-location displaying area 62 ata top right corner thereof in which a location of the pixel-displayingarea 60 in a page of the data area is displayed. The summarized-locationdisplaying area 62 takes a format of a reduced image of a diagram shownin FIG. 34, for example. The pixel-displaying area 60 shown in FIG. 38further includes a series of letters already written on in the upperarea thereof.

A description will now be given of a case of moving the displaying area61 for writing in the pixel-displaying area 60. It should be noted thatthe pixel-displaying area 53 corresponds to the pixel-displaying area60. In addition, the displaying area 54 for writing corresponds to thedisplaying area 61 for writing. The writing input device 1 initiallychecks whether the moving-area selecting key has been pressed at thestep S1401 of FIG. 37. If the writing input device 1 determines at thestep S1401 that the moving-area selecting key has been pressed, thewriting input device 1 proceeds to the step S1402, and checks whether aselected area is the pixel-displaying area 60 of the LCD 8. If it isdetermined at the step S1402 that the displaying area 61 for writing hasbeen selected instead of the pixel-displaying area 60, the writing inputdevice 1 further checks whether one of the up, down, left and rightbuttons of the direction-specifying key has been pressed at a stepS1405. If it is determined at the step S1405 that the one of the buttonshas been pressed, at a step S1406, the writing input device 1 moves aframe on the LCD 8 indicating the displaying area 61 for writing by 30pixels to a direction specified by use of the direction-specifying key,and again displays the displaying area 61 for writing at a new location.In a case that an instruction to move the displaying area 61 for writingto the right (left) has been inputted when the number of pixelsremaining on the right (left) side of the displaying area 61 in thepixel-displaying area 60 is less than 30 pixels, the displaying area 61is moved by the number of pixels remaining. FIG. 39 is a diagram showingthe displaying area 61 for writing that has been moved to the left edgeof the pixel-displaying area 60 from the position shown in FIG. 38 bythe left button of the direction-specifying key being pressed for fourtimes.

According to the eighteenth embodiment, an area corresponding to therecording area can be moved freely in a page of the data area, therebyimproving operability of the writing input device 1.

A description will now be given of a nineteenth embodiment of thepresent invention. In a case of using a lens for the left camera unit 2and the right camera unit 3, a larger distance from one of the leftcamera unit 2 and the right camera unit 3 produces a lower resolution ofreading coordinates. Thus, the nineteenth embodiment provides a methodof preventing deterioration of an image caused by lack of written databy executing an interpolation process against the written data in anarea where the resolution of reading coordinates is low. Additionally,the nineteenth embodiment provides a method of making pixel density andimage quality of written data obtained by a writing implement moving acertain distance substantially even throughout the entire recordingarea, by executing an interpolation process or a decimation processproperly based on a resolution of reading coordinates at a writingposition of the writing implement.

In the nineteenth embodiment, the writing input device 1 manages a pageof written data in the number of pixels corresponding to 100 dpi, thatis, the 864-pixel width by the 1140-pixel length. Since the size of therecording area corresponds to 1/16 of the size of a page of the writtendata as shown in FIG. 28, coordinates of the writing position aredesirably read at 25 dpi in the recording area.

In a case that the number of pixels read by a CMOS image sensor providedin each camera unit in the horizontal direction (a direction parallel toa recording surface of an image photographed by each camera unit) is 640pixels, the resolution of reading coordinates at the points R and S inthe A4-sized recording area shown in FIG. 23 is about 28 dpi.Additionally, the resolution of reading coordinates at the center of therecording area is about 56 dpi. In other words, the closer to the leftcamera unit 2 and the right camera unit 3 the writing position is, themore densely coordinates of the writing position are read by the CMOSimage sensors 16 and 21. On the contrary, the farther from the leftcamera unit 2 and the right camera unit 3 the writing position is, themore roughly the coordinates of the writing position are read by theCMOS image sensors 16 and 21. However, a resolution of readingcoordinates throughout the entire recording area only needs to be higherthan 25 dpi. Thus, the decimation process is executed on the writtendata, that is, a series of coordinates of the writing position,throughout the entire recording area. The closer to the left camera unit2 and the right camera unit 3 the writing position of the writingimplement is, the higher the resolution of reading coordinates is. Thus,the closer to the left camera unit 2 and the right camera unit 3 thewriting position is, the higher a rate of decimation should be. On theother hand, the farther from the left camera unit 2 and the right cameraunit 3 the writing position is, the lower the resolution of readingcoordinates is. Thus, the farther from the left camera unit 2 and theright camera unit 3 the writing position is, the lower the rate ofdecimation should be. Written data whose size is adapted to 1/16 of thesize of a page (216 pixels by 285 pixels) is obtained because of theabove-described execution of the decimation processes.

A description will now be given of a method of calculating a resolutionof reading coordinates in the recording area. FIG. 40 is a diagram usedfor describing a method of calculating the resolution of readingcoordinates at a point T in the recording area. FIG. 40 shows the leftcamera unit 2, the right camera unit 3, the mirror 14 and the mirror 19.An image of a writing implement located at the point T is obtained bylight reflected at around the center of the mirrors 14 and 19. However,it is assumed that the centers of the wide-angle lenses 15 and 20 arelocated respectively at points P and Q that are corners of the recordingarea for the purpose of simplifying the description. Coordinates of thepoints P and Q are respectively (0, 0) and (215, 0) since a line PQ is awidth of an A4-sized recording area.

A point T1 is provided in the recording area so that image-formedlocations of the points T and T1 are at pixels adjacent to each other onthe CMOS image sensor 16 of the left camera unit 2. Similarly, a pointT2 is provided in the recording area so that image-formed locations ofthe points T and T2 are at pixels adjacent to each other in the CMOSimage sensor 21 of the right camera unit 3. An angle θ corresponds, to aphotographing range of each pixel on the CMOS image sensors 16 and 21,and depends on a view angle of each of the left camera unit 2 and theright camera unit 3. Lengths of lines PT, QT, TT1 and TT2 are named k1,k2, L1 and L2 respectively. An angle formed by lines PT1 and TT1, and anangle formed by lines QT2 and TT2 are set to 90 degrees. In addition,coordinates of the point T are set to (x, y). Accordingly, values oflengths k1 and k2 are obtained as follows.k1=√(x ² +y ²)  EQ15k2=√((215−x)² +y ²)  EQ16

Subsequently, values of lengths L1 and L2 can be obtained from thefollowing equations.L1=k1*sin θ  EQ17L2=k2*sin θ  EQ18

The coordinates (x, y) of the point T obtained from the equations EQ1through EQ4 are expressed in millimeters so that the lengths L1 and L2obtained from the equations EQ17 and EQ18 are expressed also inmillimeters. Thus, a unit of the lengths L1 and L2 should be convertedfrom a millimeter to an inch by use of a fact that one-millimeter isequal to about 0.03937 inch. Subsequently, a resolution Rdiv_L ofreading coordinates by the left camera unit 2 and a resolution Rdiv_R ofreading coordinates by the right camera unit 3 at the point T areobtained respectively by dividing one inch by the length L1 expressed ininches and by the length 2 expressed in inches.Rdiv_(—) L=1/(L1×0.03937)  EQ19Rdiv_(—) R=1/(L2×0.03937)  EQ20

Since the point T is provided in the recording area so that the value ofk1 is greater than the value of k2 as shown in FIG. 40, the value of L1obtained from the equation EQ17 is greater than the value of L2 obtainedfrom the equation EQ18. Thus, the value of the resolution Rdiv_Robtained from the equation EQ20 is greater than the value of theresolution Rdiv_L obtained from the equation EQ19.Rdiv_(—) R>Rdiv_(—) L  EQ21

As seen in the equation EQ21, the resolution of reading coordinates bythe right camera unit 3 is higher than the resolution of readingcoordinates by the left camera unit 2.

As described above, the resolution of reading coordinates by the leftcamera unit 2 is different from the resolution of reading coordinates bythe right camera unit 3 at any point except a point where the values ofk1 and k2 are identical. Accordingly, the writing input device 1calculates a vector between each coordinate data belonging to a seriesof the coordinate data, and decides to use one of resolutions of readingcoordinates by the left camera unit 2 and by the right camera unit 3based on a direction of the vector between each set of coordinates, forexecuting the decimation process on written data. For example, theresolution of reading coordinates by the left camera unit 2 is used whena user is writing data in a top-right direction as shown in FIG. 41A. Onthe other hand, the resolution of reading coordinates by the rightcamera unit 3 is used when the user is writing data in a bottom-rightdirection. Similarly, the resolution of reading coordinates by the rightcamera unit 3 is used when the user is writing data in a top-leftdirection. The resolution of reading coordinates by the left camera unit2 is used when the user is writing data in a bottom-left direction.

FIG. 42 is a flowchart describing steps performed by the writing inputdevice according to the eighteenth embodiment of the present invention.The writing input device 1 initially checks whether coordinates havebeen inputted to the recording area at a step S1501. If it is determinedat the step S1501 that the coordinates have been inputted to therecording area, the writing input device 1 inspects a direction oftaking in coordinate data at a step S1502. If the direction of taking incoordinate data is the top-right or bottom-left direction, the writinginput device 1 obtains the resolution of reading coordinates by the leftcamera unit 2 at a step S1503. If the direction of taking in coordinatedata is the bottom-right or top-left direction, the writing input device1 obtains the resolution of reading coordinate data by the right cameraunit 3 at a step S1506. At a step S1504 following the steps S1503 andS1506, the writing input device 1 executes the decimation process onwritten data inputted thereto at a fixed rate of decimation thatcorresponds to the resolution of reading coordinates by a camera unitselected at the step S1502 so that a pixel density of the written databecomes 100 dpi. Subsequently, at a step S1505, the writing input device1 adds the written data that has been decimated at the step S1504 to arecording area that corresponds to the written data in a page of writtendata as shown in FIG. 29. In a case that there is a data-storingoperation, the writing input device 1 stores a renewed page as a file inthe flash memory 26. It should be noted that a resolution of readingcoordinates during a data-writing process in the recording area variesaccording to coordinates of the writing position. Accordingly, theentire recording area may be equally divided into a plurality of areas,wherein a resolution of reading coordinates in each area may be set to asingle value, for example, a value at the center of the area.

A description will now be given of a method of executing aninterpolation process on written data. The above-described processesshown in FIG. 42 may be applied to an interpolation process instead of adecimation process. In such case, the writing input device 1 obtains avector between each set of coordinates of written data, and decideswhich of the left camera unit 2 or the right camera unit 3 is to be usedfor calculating a resolution of reading coordinates. In this method ofexecuting an interpolation process on written data, the writing inputdevice 1 manages a page of written data in a size of a 1728-pixel widthby a 2280-pixel length, which corresponds to 200 dpi. Since the size ofthe recording area is 1/16 of a page of the written data, the writinginput device 1 needs to read coordinates of a writing position at 50 dpithat is ¼ of the resolution of a page of the written data. However, in acase that the number of pixels read by the CMOS image sensors 16 or 21in the horizontal direction is 640 pixels, the resolution of readingcoordinates at the points R and S in the A4-sized recording area shownin FIG. 23 is about 28 dpi so that the CMOS image sensors 16 and 21cannot read the coordinates of the points Rand S at 50 dpi.

Accordingly, the writing input device 1 generates written data at 50 dpiby executing the interpolation process on written data, that is, aseries of coordinates read by the CMOS image sensors 16 and 21, in acase that the resolution of reading coordinates is lower than 50 dpi. Aspline-curve method and a Bezier-curve method are used in theinterpolation process executed on the written data. The spline-curvemethod is a method of interpolating coordinates at a fixed interval on acurve after obtaining the curve that includes all the coordinatesobtained by the CMOS image sensors 16 and 21 thereon. The Bezier-curvemethod is a method of interpolating coordinates at a fixed interval on acurve after obtaining the curve that includes first and last coordinatesprovided in a series of coordinates of a writing position thereon byusing coordinates located between the first and last coordinates onlyfor deciding a shape of the curve. In other words, the first and lastcoordinates are on a Bezier curve. However, other coordinates locatedbetween the first and last coordinates are not necessarily on the Beziercurve. As described above, the writing input device 1 executes theinterpolation process on written data in a case that a resolution ofreading coordinates is less than 50 dpi according to the writingposition. Alternatively, the writing input device 1 executes thedecimation process on written data in a case that a resolution ofreading coordinates is higher than 50 dpi. Subsequently, the writinginput device 1 adds the written data having its pixel density being 200dpi to a recording area corresponding to the written data in a page ofwritten data.

A description will now be given of a twentieth embodiment of the presentinvention. The twentieth embodiment provides a method of implementingthe present invention by use of software. FIG. 43 is a block diagramshowing a system wherein the present invention is implemented by use ofsoftware. The system shown in FIG. 43 includes a coordinate-inputtingunit 70, a CPU 71, a memory 72, an image-displaying device 73, a CD-ROMdevice 74, a CD-ROM 75, and a communication device 76. Thecoordinate-inputting unit 70 includes the left camera unit 2 and theright camera unit 3 shown in FIG. 3. The CPU 71 executes the steps andfunctions that have been described in each of the previous embodimentson an image photographed by the left camera unit 2 and the right cameraunit 3. Subsequently, the CPU 71 displays depiction data on theimage-displaying device 73, and stores written data in the memory 72. Aprogram executing the above-described processes is stored in a recordingmedium such as the CD-ROM 75. The program stored in the recording mediumis read by the CD-ROM device 74, and is installed in the system, therebyexecuting the functions described in the above-described embodiments.The program may be supplied to the system by being downloaded from aserver and the like through the communication device 76 and a network77.

As described above, the present invention enables use of a desiredwriting implement and a desired recorded material whose recordingsurface is a plane for a writing input device. Additionally, accordingto the present invention, the writing input device can record data in arecording medium in real time while the data is being written by use ofa writing implement on the recording surface. Additionally, the writinginput device can create and manage a page of written data by dividingthe page into a plurality of areas, and then by assigning each of theareas to a recording area even in a case that a resolution of readingcoordinates is low in the recording area. Additionally, the writinginput device can control a pixel density of the written data to be eventhroughout the entire recording area. Furthermore, the writing inputdevice can control displaying the recording area on a LCD for improvingoperability of writing data in the recording area.

The above description is provided in order to enable any person skilledin the art to make and use the invention and sets forth the best modecontemplated by the inventors of carrying out the invention.

The present invention is not limited to the specially disclosedembodiments and variations, and modifications may be made withoutdeparting from the scope and spirit of the invention.

The present application is based on Japanese Priority Application No.11-369699, filed on Dec. 27, 1999, the entire contents of which arehereby incorporated by reference.

1. An interactive input system, comprising: at least two camerasconfigured to be associated with a display writing surface, the at leasttwo cameras configured to capture images of a region adjacent to thedisplay writing surface from different locations and having at leastpartially overlapping fields of view; and at least one computer readablemedium having program instructions configured to cause at least oneprocessing structure to: determine whether an object extracted from thecaptured images is a writing implement by determining, when a pluralityof objects are extracted from the image, that one of the plurality ofobjects that satisfies a prescribed condition is the writing implement;detect an outline of the writing implement from the captured images whenthe writing implement intersects a boundary of the captured images;calculate an input position based on the outline of the writingimplement intersecting the boundary of the captured images; and inputthe information representing the input position on the writing surfaceindicated by the writing implement.
 2. The interactive input system ofclaim 1, wherein the boundary of the captured images corresponds to aplane of the writing surface.
 3. The interactive input system accordingto claim 2, wherein the at least one computer readable medium hasprogram instructions configured to cause the at least one processingstructure to: observe movement of the writing implement approaching theboundary of the captured images; and store the calculated input positionof the writing implement when the writing implement has stopped in aperpendicular direction to the plane.
 4. The interactive input systemaccording to claim 2, wherein the at least one computer readable mediumhas program instructions configured to cause the at least one processingstructure to define the calculated input position as valid informationif the input position is located in a fixed area on the plane.
 5. Theinteractive input system according to claim 4, wherein the fixed areacomprises an area selected from a plurality of areas whose shapes andsizes are different from each other.
 6. The interactive input systemaccording to claim 4, wherein the fixed area comprises an area that canbe set to a fixed shape.
 7. The interactive input system according toclaim 6, wherein the fixed area comprises one of (i) a closed areaformed by a track of the input position of the writing implement on theplane, and (ii) an area formed by the track and a width of apredetermined area.
 8. The interactive input system according to claim2, wherein the at least one computer readable medium has programinstructions configured to cause the at least one processing structureto define the calculated input position as invalid information as wellas signaling the invalid information if the input position is notlocated in a fixed area.
 9. The interactive input system according toclaim 1, wherein the at least one computer readable medium has programinstructions configured to cause the at least one processing structureto display an image based on the calculated input position.
 10. A methodcomprising: capturing images of a region adjacent to a display writingsurface from different view angles; processing the captured images todetermine whether an object extracted from the captured images is awriting implement by determining, when a plurality of objects areextracted from the image, that one of the plurality of objects thatsatisfies a prescribed condition is the writing implement; processingthe captured images to detect an outline of the writing implement whenthe writing implement intersects a boundary of the captured images;calculating an input position based on the detected outline of thewriting implement intersecting the boundary of the acquired images; andinputting the information representing the input position on the writingsurface indicated by the writing implement.
 11. The method of claim 10,wherein the boundary of the captured images corresponds to a plane ofthe writing surface.
 12. The method of claim 11, further comprising:observing movement of the writing implement approaching the plane; andstoring the calculated input position of the writing implement when thewriting implement has stopped in a perpendicular direction to the plane.13. The method of claim 11, further comprising defining the calculatedinput position as valid information if the calculated input position islocated in a fixed area on the plane.
 14. The method of claim 13,wherein the fixed area comprises an area selected from a plurality ofareas whose shapes and sizes are different from each other.
 15. Themethod of claim 13, wherein the fixed area comprises an area that can beset to a fixed shape.
 16. The method of claim 15, wherein the fixed areacomprises one of (i) a closed area formed by a track of the calculatedinput position of the writing implement, and (ii) an area formed by thetrack and a width of a predetermined area.
 17. The method of claim 11,further comprising defining the input position as invalid informationand signaling the invalid information if the calculated input positionis not located in a fixed area.
 18. The method of claim 10, furthercomprising displaying an image based on the calculated input position.19. A non-transitory computer readable medium having programinstructions configured to cause at least one processing structure to:determine whether an object extracted from received images of a regionadjacent to a display writing surface is a writing implement bydetermining, when a plurality of objects are extracted from the image,that one of the plurality of objects that satisfies a prescribedcondition is the writing implement; detect an outline of the writingimplement from the received images when the writing implement intersectsa boundary of the received images; calculate an input position based onthe detected outline of the writing implement intersecting the boundaryof the received images; and input the information representing the inputposition on the writing surface indicated by the writing implement. 20.The non-transitory computer readable medium of claim 19, wherein theboundary of the received images corresponds to a plane of the writingsurface.